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Abstract —When developing mobile apps, programmers rely 
heavily on standard API frameworks and libraries. However, 
learning and using those APIs is often challenging due to the 
fast-changing nature of API frameworks for mobile systems, the 
complexity of API usages, the insufficiency of documentation, 
and the unavailability of source code examples. In this paper, we 
propose a novel approach to learn API usages from bytecode 
of Android mobile apps. Our core contributions include: i) 
ARUS, a graph-based representation of API usage scenarios; 
ii) HAPI, a statistical, generative model of API usages; and iii) 
three algorithms to extract ARUS from apps’ bytecode, to train 
HAPI based on method call sequences extracted from ARUS, and 
to recommend method calls in code completion engines using 
the trained HAPI. Our empirical evaluation suggests that our 
approach can learn useful API usage models which can provide 
recommendations with higher levels of accuracy than the baseline 
n-gram model. 

Keywords—Statistical model, API usage, mobile apps 
I. Introduction 

Millions of mobile apps (i.e. software applications for 
mobile devices like smartphones and tablets) have been de¬ 
veloped and made available to mobile users. Due to the fierce 
competition, those mobile apps often have very short time-to- 
market and upgrade cycles, thus, requiring short development 
time. To address this requirement, programmers often rely 
heavily on API application frameworks and libraries such as 
Android and iOS frameworks, Java APIs, etc. For example, an 
Android app might make 8-42% of its external dependencies 
to Android APIs and 7-68% to Java APIs (T). 

Learning and using APIs is often challenging due to several 
reasons. First, a framework often include large numbers of 
API elements (i.e. class or method). For example. Android 
application framework contains over 3,400 classes and 35,000 
methods, clustered in more than 250 packages [2J. Moreover, 
typical API usage scenarios often include several API elements 
and follow special rules, e.g. for pre- and post-conditions 
or for control and data flows 0-0- Unfortunately, API 
documentation is often insufficient. For example, the Javadoc 
of a class often contains only descriptions of its methods 
and rarely has code examples on the usage of its objects 
and methods [6). Documentation and code examples for API 
usages involving several objects are often non-existed. 

The situation is even more difficult for learning APIs of 
mobile frameworks. First, due to the fast development of 
mobile devices and the strong competition between vendors, 
those frameworks are often upgraded quickly and include very 
large changes. For example, 17 major versions of Android 


framework have been released within five years, making nearly 
100,000 changes to its API methods |2). More severely, mobile 
apps are often closed-source, i.e. their source code is not 
publicly available. Thus, finding and learning code examples 
from existing mobile app projects would be difficult. 

Due to those reasons, the current best API learning ap¬ 
proach for mobile app programmers would be looking for on¬ 
line tutorials and discussions on developer forums like Stack- 
Overflow.com. However, this approach is rather ineffective 
and time-consuming, thus, could severely affect programming 
productivity. In addition, it is also error-prone because online 
code examples might not be up-to-date to the latest changes of 
the mobile API frameworks. When programmers do not learn 
from accurate materials, they could use the APIs improperly, 
causing errors and defects and thus, reducing the quality and 
the usability of their apps 0, 

In this paper, we introduce a novel approach, SALAD, to 
address those problems. Standing for “Statistical Approach 
for Learning APIs from DVM bytecode”, our approach has 
three distinctive characteristics. First, to address the problem of 
insufficient documentation and source code examples, SALAD 
learns API usages from bytecode of Android mobile apps, 
which are publicly available in very large quantities (up to 
millions of apps). Second, SALAD can learn complex API 
usages consisting large numbers of objects and methods. 
Finally, SALAD can automatically generate recommendations 
for incomplete API usages, thus could reduce the chance of 
API usage errors and improve code quality. 

Let us discuss SALAD in more details. The most important 
component of SALAD is HAPI, standing for “Hidden Markov 
Model of API usages”. A HAPI is an Hidden Markov Model 
(HMM) |7 | which models the usages of one or multiple API 
objects. As a statistical model, it has several internal states, 
each has a probability to be selected as the starting state. 
When a HAPI is in a state, it can generate a method call 
and change to another state with specified probabilities. We 
have designed an algorithm to train a HAPI (i.e. inferring its 
internal states and the associating probabilities) from a given 
collection of method call sequences involving the API objects 
it models. Another algorithm is designed to use a trained HAPI 
to compute the generating probabilities of several method call 
sequences to rank those sequences. This ranking result is 
used in the recommendation task of SALAD. More details on 
HAPI and the training/ranking algorithms will be discussed in 
Section III and IV. 

To extract and generate API method call sequences from 


bytecode for training HAPI, we develop another conceptual 
model named ARUS, standing for ‘‘Abstract Representation 
of Uasge Scenario”. An ARUS is a graph-based model in 
which the nodes represent method invocations and data objects 
and the edges in the ARUS indicate control and data flows 
between those methods and objects. With that design, an 
ARUS can represent an usage scenario including many objects 
and methods and complex control and data flows between 
them. We have designed and implemented in SALAD an 
algorithm to extract ARUS models directly from Android apps’ 
bytecode. More details on ARUS and this algorithm will be 
discussed in Section III and IV. 

To evaluate the usefulness and effectiveness of SALAD, we 
have conducted several experiments comparing the accuracy 
in recommending API usages of HAPI and the baseline n- 
gram model which is widely used in most prior studies. 
The experiments show two important results. First, HAPI can 
provide API usage recommendations with a high level of 
accuracy. For example, it can predict and recommend the next 
API call for a given method sequence with a top-3 accuracy 
of more than 70% and a top-10 accuracy of around 90%. 
Second, HAPI consistently outperforms n-gram model with 
a difference of around 10%. 

In summary, the core contributions of this paper include: 

1. ARUS, a graph-based representation of API usage scenarios; 

2. HAPI, a statistical, generative model of API usages; and 

3. Three algorithms to extract ARUS from apps’ bytecode, to 
train HAPI based on method call sequences extracted from 
ARUS, and to recommend method calls in code completion 
engines using the trained HAPI. 

4. An empirical evaluation for the accuracy and usefulness of 
SALAD 

The remaining of this paper is organized as the following. 
Section II presents some examples of Android APIs and their 
usages in practice. Section III introduces our two main concep¬ 
tual contributions: HAPI and ARUS. Section IV discusses the 
overall architecture of SALAD and its main algorithms. Our 
empirical evaluation is presented in Section V. We discussed 
related work in Section VI and conclude this paper in the last 
section. 


II. Motivation 

In this section, we discuss an example to demonstrate the 
challenges of learning APIs in programming. Assume that a 
developer wants to program his app to record audio and video. 
He thinks of class MediaRecorder in the Android application 
framework but does not know how to write the code to 
initialize a MediaRecorder object and calls relevant methods. 
In other words, he has to learn the usages of this API object. 

One way to learn such API usages is to read object state di¬ 
agrams. Figure [T] illustrates the state diagram of MediaRecorder, 
which is reproduced from Android Developer websitcj^] As 
seen in the figure, the state diagram of an object is a finite 
state machine in which each node represents a state of the 
object and each edge represents a method call. We learn from 
the state diagram that a MediaRecorder object has 7 states 
during its lifetime and several methods can be called to change 
from one state to another. For example, if the object is in 
state Prepared and method start() is called, it will change to 

1 http://developer.android.com/reference/android/media/MediaRecorder.html 
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setAusioSourceO 

setVideoSourceO 
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setVideoSourceO 


setOutputFormatO 


C Recording 


setAudioEncoderO 
setVideoEncoderO 
setOutputFileO 
setVideoSizeO 
setVideoFrameRateO 
setPreviewDisplayO 
Fig. 1: The MediaRecorder State Diagram 


C Prepared 


protected void startRecording(String file) { 

MediaRecorder recorder = new MediaRecorder(); 

recorder.setAudioSource(MediaRecorder.AudioSource.MIC); 

recorder.setOutputFormat(MediaRecorder.OutputFormat.THREE_GPP); 

recorder.setAudioEncoder(MediaRecorder.AudioEncoder.AMR_NB); 

recorder.setOutputFile(file); 

try { 

recorder.prepareO; 

} catch (HlegalStateException e) { 
e.printStackTrace(); 

} catch (lOException e) { 
e.printStackTrace(); 

} 

recorder.start(); 

} 

Fig. 2: Source code examples of using MediaRecorder objects 

state Recording. Although useful, state diagrams are often non- 
existed for most API objects. In addition, it is difficult to infer 
from the diagram the usages for a typical or popular task, such 
as the method sequences to set up a MediaRecorder for audio 
or video recording. 

A more popular way to learn API usages is from reading 
real-world code examples. For example. Figure [2] provides 
an example using a MediaRecorder object to record audio. 
Although more illustrative, this example also shows that this 
usage scenario of a MediaRecorder object for recording audio 
is rather complex. It involves 7 method calls, two exception 
handler, and several constants. More severely, for mobile API 
framework, code examples of most API usages are often 
unavailable or insufficient. Unlike in traditional software devel¬ 
opment where massive open source repositories are available, 
when developing mobile apps, developers often publish their 
apps to only online app stores. In other words, most mobile 
apps are closed-source, i.e. their source code is not available 
online for code search. 

From this examples we see that programmers can learn 
API usages from state diagrams and code examples to use 
them correctly. However, those learning materials are often 
complex, unavailable, or insufficient, especially for mobile API 
framework. 

To address these challenges, in this paper, we propose 
HAPI (Hidden Markov Model of API usages), a statistical 
model for API usages that learns “probabilistic” state diagrams 
from a vast amount of publicly available bytecode. Our idea is 
based on an assumption that usage patterns specified by state 
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Fig. 3: The HAPI model to represent usage patterns of Medi- 
aRecorder object 

diagrams exhibit on code examples. HAPI is adopted from 
Hidden Markov Models [7J, a probabilistic generative model 
for modeling sequences with “hidden” states. 

III. Approach 

Figure [3]illustrates the “probabilistic” state diagram (HAPI) 
representing the usages of MediaRecorder object learned by our 
approach. As seen in the figure, each node in HAPI represents 
a state of the object as in the state diagram. But different from 
state diagrams, each state in HAPI has a probability to be 
selected as the starting state 7r and a calling distribution which 
specifies the probability of calling a particular method at that 
state. An directed edge from state x to state y has a weight 
representing the probability of switching from state x to state 
y after calling a method. In the figure, to simplify the drawing, 
we only show edges with significant weights. 

HAPI has several advantages over state diagrams. First, 
the state diagram only specifies the general rules for using 
objects but not specify the common usages. Once learned 
from code examples, HAPI could infer common usages of an 
object by searching for the method sequences that have highest 
probability. Second, HAPI could be used to recommend API 
usages when developers are programming. For example, given 
a sequence of API method calls, it can predict the most likely 
next method call. Let us describe it in details. 

A. Hidden Markov Models for learning API usages 

In this section, we present HAPI, a statistical model for 
API usages, which is adopted from Hidden Markov Models 
[7j. HAPI is a generative probabilistic model that describes 
the process of generating API method sequences of an API 
object (set of objects). To describe the model, we denote the 
API method invocation at time t by the random variable Y t . 
Y t is a variable follows the discrete distribution and t is an 
integer-value index. 

HAPI has two defining properties. First, HAPI has sev¬ 
eral internal states that represent states of the object (set of 
objects). Each state has a calling distribution which specifies 


the probability of calling a particular method on the state, 
and a transition distribution which specifies the probability 
of changing from that state to a particular state after calling 
a method. HAPI assumes that the method invocation on the 
object at time t is sampled from the calling distribution of 
state q t which is hidden from observers. We could think that 
the hidden state q t is the current state of the object time t. 
Second, HAPI assumes that the state q t satisfies the Markov 
property, that is, given the value of q t _ i, the probability of 
changing to q t is independent of all the states prior to t — 1. In 
other words, the state at some time encapsulates all we need 
to know about the future of the generation process. 

For a formal description, we will use the following nota¬ 
tions to characterize a HAPI: 

• Q = {<Zi, Q 2 , •••) Qk }i the set of K hidden states of an 
API object (set of objects) 

• V = {iq, i> 2 ,..., Vm}, the set of API methods asso¬ 
ciate with the object (set of objects). 

• A = {a,ij,a,ij = P(qj at t + l\qi at f)}, the state 
transition matrix, where the state transition coefficients 
satisfy ^. <Hj = 1, aij is transition probability from 
state q.i to state qj 

• B = {bj(m)},bj(m) = P{v m at t\qj at t), bj is the 
calling distribution in state q 3 , bj(m ) is the probability 
of calling method v rn in the state q 3 . 

• 7T = {77^}, 7Ti = P(qi at t = 1), the initial state 
distribution, where Hi is the probability of that state q, 
is the beginning state to generate a method sequence. 

Given the HAPI model of an object (a set of objects) with 
parameters A = (A, B, ir), the generative process of an method 
sequence Y = (yi, j/ 2 , Z/t) is as follows: 

1. Sample an initial state i\ from the initial state distribution 
7r and set t = 1 

2. Sample an method call y t from the calling distribution of 

it, b it (m) 

3. Sample z t+1 from the transition distribution of i t , a,i t i t+1 

4. Set t = t + 1 and return to step 3 if t < T; otherwise end. 

Using HAPI to model API usages, in our work, we are 
interested in 1) how to learn HAPI model of an API object 
(set of objects), 2) how to make used of learned HAPI models. 
For the first problem, we present an algorithm to train a HAPI 
(estimating its parameter A = (A,B, tt)) given collection of 
method sequences involvings the API object (set of objects). 
For the second problem, in our work, we focus on using HAPI 
for API usage recommendation. We design an algorithm to use 
the trained HAPI to compute the generating probabilities of 
several method sequences to rank those sequences and use the 
ranking for recommendation task. 

B. Abstract Representation of Usage Scenario 

Input to train HAPI model is API method sequences 
associate with single object or a set of objects involves in an 
usage. To extract such sequences, we propose a novel graph- 
based representation called Abstract Representation of Usage 
Scenario (ARUS). ARUS is a labeled, directed graph represent¬ 
ing an usage scenario in one execution path including many 
objects, methods, complex control, and data flows between 
them. 

Definition 1 (ARUS): ARUS is a directed graph that has 
two types of nodes and edges. Object nodes represent objects 






















that are involved or created during the execution. Action nodes 
are any actions that performs on object nodes. Action nodes 
could be object instantiations, method calls, data field accesses, 
or other operations. Control edges represent temporal orders 
between action nodes. Data edges describe data-dependency 
between object and action nodes. 

In ARUS, each object created or involved during the 
execution is represented as an object node. We also treat 
primitive variables as object nodes. Action nodes represent any 
action that is performed on object nodes. Action nodes could 
be object instantiations, method calls, data field accesses of one 
object, or other operations. Object nodes are labeled by class 
names (object nodes represent primitive variables are labeled 
by types). Action nodes of types object instantiations, method 
calls, or data field accesses are labeled as “C.m” with C is its 
class name and m is the method (or field) name. Other action 
nodes that represent operations are labeled as the name of the 
operation. 

The control edges of ARUS are used to represent the 
temporal orders between action nodes. A control edge from 
an action node A to action node B means that A is executed 
before B in the execution path. Because ARUS is defined for 
each execution path, thus, there is only one temporal order 
between action nodes, which is represented by a set of control 
edges between action nodes. The data edges indicate the data 
dependencies between data nodes and action nodes. A data 
edge from object node A to action node B means that A is a 
parameter of the action that B represents. A data edge from 
an action node B to data node A means the action B returns 
object node A as a result. 

Figure [8] shows two ARUSs of a method. Rectangle nodes 
are action nodes, while object nodes are represented as round 
rectangle nodes. Solid arrows represents control edge between 
action nodes and dashed arrows represent data edges. 

Definition 2 ( Usage-dependency): A set of object nodes is 
said to have usage-dependency if there is at least one action 
node that have data-dependent with all the object nodes in the 
set. 

There are two main advantages of representing usage 
scenario using ARUS. First, we could easily extract action 
sequences associates with one or a set of objects. For a object 
node (set of object nodes), we identify all action nodes that 
have data-dependency with that object node (set of object 
nodes) based on data edges. The action sequence follows 
that temporal order between action nodes. In case we want 
to extract API method sequences we filter out all non API 
action nodes in action sequences. The second advantage of 
ARUS is that we could define whether multiple object nodes 
join in an usage or they are used in several usages separately. 
We define a set of object nodes usage-dependency if there is 
at least one action node that have data-dependency with all 
the object nodes in the set. By defining usage-dependency we 
could model API usages that involves several objects. 

IV. System Description 

Our approach for learning API usages consists six major 
components: app crawler, .dex file parser, ARUS analyzer, 
sequence extractor, HAPI learner, and API usage recommenda¬ 
tion. Figure [4] shows an overview of all the components in the 
system. Our approach consists of three main phases. In app 
processing phase, the app crawler downloads (free) Android 


applications from the Google Play Store and extract the .dex 
files (Dalvik Execution Format) that contain bytecode of those 
apps to store in local repository. The .dex file parser reads .dex 
files and construct Control Flow Graph (CFG) for bytecode 
of each method. In analyzing phase, ARUS analyzer takes 
a CFG as an input and produces ARUS for each execution 
path in the CFG. Sequence extractor uses these produced 
ARUSs and extracts API method sequences of one or multiple 
objects. In the learning phase, the HAPI learner uses the API 
method sequences that have been extracted to learn API usage 
models. For evaluation, we used our HAPI models for API 
usage recommendation. Next, we present the details for each 
component in the system. 

A. App Crawler 

In SALAD, we first built a crawler to downloads applica¬ 
tions from Google Play Store. We developed our app crawler 
based on the google-play-crawler project |8j. The crawler 
simulates an android devices and it has the same accessibility 
to the store as a normal phone such as searching for apps, 
browsing categories and apps, app information, downloading 
free apps, etc. On Google Play Store, each category contains 
top lists of apps. We downloaded all the top free/new free apps 
in all categories. In each application package (.apk files), we 
only kept the .dex files that contain all the bytecode of the 
applications. 

B. .dex File Parser 

The next step of data processing phase is parsing .dex files 
to obtain bytecode instructions of methods. For this task we 
use smali tool which is an assembler/disassembler for An¬ 
droid’s dex format. After obtaining the bytecode instructions 
for methods, to make it easier to analyze, we contruct Control 
Flow Graph (CFG) from those instructions. 

Dalvik Virtual Machine. Android applications are devel¬ 
oped in Java, but unlike a normal Java application, which 
is compiled into Java Virtual Machine (JVM) class files and 
executed in JVM, Android applications are compiled to Dalvik 
bytecode and are executed in Dalvik Virtual Machine (DVM). 
DVM is a virtual machine that provides runtime environment 
for application in Android platform. Dalvik bytecode of all 
methods implemented in an Android application is stored in a 
single .dex file inside the application package. 

DVM is a register-based virtual machine. DVM uses a 
set of 32-bit registers to hold bit values (such as integers 
and floating point numbers) and object references. Frames 
(activation records) of a method are fixed in size which consists 
of a particular number of register. These registers are used to 
store local variables, parameters, return values, and temporary 
values instead of using stacks as in Java Virtual Machine. The 
N arguments to a method are stored in the last N registers 
of the method’s frame. Instance methods are passed a this 
reference as their first argument. 

Figure[6]shows Dalvik bytecode for the readTextFile method. 
The method is allocated 7 registers v0-v6 to executes instruc¬ 
tions. Registers 5 and v6 store parameters, v5 is the object that 
the method is being invoked (because startRecording is an in¬ 
stance method), v6 is a string contains file name. The registers 
v0-v4 are used to store local or temporary variables. Instructions 
in Dalvik bytecode operate on registers. For example, mul-int 
v2,v5,v3 instruction multiples the values of registers v2 and v5 



Fig. 4: The overview of SALAD 


and stores the result to v3, or newinstance vl, Ljava/io/FileReader; 
creates a new FileReader object and returns the reference of 
that object to register vl. 

Control Flow Graph. We construct Control Flow Graph 
(CFG), which is a representation of bytecode instructions. 
A node in the constructed CFG contains a single bytecode 
instruction and an edge is the control flow between the two 
instructions. There are two types of nodes in CFG. Control 
nodes represent control instructions in bytecode, e.g. if, return, 
throw, or goto instructions. Other instructions are normal nodes. 
Normal nodes in a CFG only have one out-going edge points to 
the next instruction while control nodes could have several out¬ 
going edges (if, switch nodes) or do not have any (return nodes). 
Techniques used for constructing CFGs are quite standard. 
First, we create all nodes in the CFG, each one corresponds 
to an instruction in the instruction list. Then, for each node, 
we use offsets to identify nodes that are executed after it and 
add edges from the current node to those nodes. 


1 function BuildARUS(Method M) 

2 CFG = BuildCFG(M) 

3 A = 0 //list of ARUSs 

4 St — CreateStartState(C.FG, Af) 

5 F = 0 

6 Push(F, S t ) 

7 while F ^ 0 

8 S = Pop (F) 

9 SN = GetStartNode(S) 

10 while SN is not a control node 

11 BuildTemporaryARUS(S, SN) 

12 AddExploredNode(S, SN) 

13 CN = GetNextNode(SJV) 

14 if SN is the return node 

15 AddARUS(A, S) 

16 else 

17 for NN g GetNextNodes(SAf) 

18 if NN g GetExploredNodes(S) 

19 S c = GetCopy(S) 

20 SetStartNode(S c , NN) 

21 Push(F, S c ) 

22 return A 


Fig. 7: Building ARUS Algorithm 


public String readTextFile(String fileName) throws lOException { 
FileReader fr = new FileReader(fileName); 
BufferedReader br = new BufferedReader(fr); 
StringBuilder strBuilder = new StringBuilderf); 

String line; 

while((line = br.readLineO) != null) { 
strBuilder.append(line); 

} 

br.closeO; 

return strBuilder.toString(); 

} 


Fig. 5: A source code example 


[0001 dc] IOManager.readTextFile:(Ljava/lang/String;)Ljava/lang/String; 

0000: new —instance vl, Ljava/io/FileReader; 

0002: invoke —direct {vl, v6}, Ljava/io/FileReader;.<init>:(Ljava/lang/String;)V 
0005: new —instance vO, Ljava/io/BufferedReader; 

0007: invoke —direct {v0, vl }, Ljava/io/BufferedReader;.<init>:(Ljava/io/Reader;)V 
000a: new —instance v3, Ljava/lang/StringBuilder; 

000c: invoke—direct {v3}, Ljava/lang/StringBuilder;.<init>:()V 

OOOf: invoke—virtual {v0}, Ljava/io/BufferedReader;.readLine:()Ljava/lang/String; 

0012: move —result—object v2 
0013: if-nez v2, 001 d 

0015: invoke—virtual {v0}, Ljava/io/BufferedReader;.close:()V 

0018: invoke—virtual {v3}, Ljava/lang/StringBuilder;.toString:()Ljava/lang/String; 

001b: move —result—object v4 
001 c: return—object v4 

OOld: invoke—virtual {v3, v2}, Ljava/lang/StringBuilder;.append:(Ljava/lang/String;)Ljava/lang/StringBuilder; 
0020: goto OOOf 


Fig. 6: Dalvik bytecode for source code example in Figure [ 5 ] 

C. ARUS Analyzer 

ARUS analyzer takes CFG as input and produce ARUSs 
which describes usage scenario for each execution path. 

1) The Algorithm: The main idea of the algorithm to 
construct ARUS is to explore all the execution path in CFG 
and build temporary ARUSs when exploring those paths. Once 
a path has been explored, it collects the ARUS that have been 
built for that path. Each CFG has a start node which are the first 
instruction and a termination node which is the return node. 
Our algorithm needs to find all execution paths from the start 
node to the termination node. One problem that the algorithm 
needs to consider is to handle loops that occur in the CFG. 
These loops represents while or for loops in source code. The 
instructions inside loops may be executed either several times 
or zero time, thus, could lead to infinity number of execution 


paths. On the other hand, considering these instructions once 
can help build usage scenario associated with loops. Therefore, 
our algorithm treats instructions inside a loop are only executed 
either once or not. In other words, a loop is considered to 
executed at most one time. 

Detailed Algorithm. Figure [7] show the pseudo-code for 
our algorithm. Input of the algorithm is the bytecode of a 
method M. The algorithm starts with creating CFG from the 
bytecode instructions using techniques described the previous 
section (line 1). Similar to depth first search algorithm, we 
maintain a stack F to store states which are frontiers to explore 
(line 4). Each state of our algorithm represents an incomplete 
execution path and contains following information: 1) start 
node: the current node in CFG when a state is pop from stack 
F, 2) explored nodes: a set of all nodes in CFG that have been 
visited 3) a temporary ARUS. The initial state St is created 
in line 3 with start node is the begin node of CFG, explored 
nodes and the temporary ARUS of this state are empty. The 
stack F is initialized with St in line 4. In the main while loop 
of the algorithm, each time, a state S is pop from F. We start 
with SN, which is the start node of S (line 8) and explore the 
path from SN to the next control node in CFG. Whenever 
SN is a normal node in CFG, we use SN to build and update 
the temporary ARUS of S (we will describe the algorithm to 
build a temporary ARUS in the next section). We then add SN 
to the set of explored nodes of S and update SN equal to the 
next node of SN in CFG (because SN is still a normal node 
in CFG, it only has one next node). After the loop from line 
9 to line 12. SN now is a control node in CFG. If SN is 
the return node of CFG then we have explored one execution 
path of CFG, the temporary ARUS of S now becomes a final 
ARUS, thus, it is added to the list A (line 14). If SN is not 
the return node, we need to consider all unexplored out going 
edges of SN. For each next node NN of SN, if NN is not 
in the set of all explored nodes of S, we copy all information 
of S into a new state S c , set NN is the start node of S c , and 
push S c into F. The algorithm terminate when the stack F 
becomes empty. Our algorithm explored all the paths of CFG 
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Fig. 8: ARUSs 

and only go through each loop at most one time. The list A 
now contains all ARUS of the method M. We next describe 
algorithm to build temporary ARUS. 

2) Building temporary ARUS: In this section, we describe 
our technique to build and update the temporary ARUS. 
Initially, before considering any instructions, we create an 
empty ARUS. For each parameter of the method we create 
a corresponding object node and add to the ARUS. When 
an instruction is considered (line 14), it create a new action 
node and add that node to the current ARUS. After adding 
the new action node, we need to update edges. We add a 
control edge from the last inserted action node to the new 
action node. For data edges, we first need to add data edges 
from object nodes that are parameters of the instruction to the 
new action node. To do that, we maintain a map M current 
to store the current mapping between registers and object 
nodes. When an action node is created, we use M current to 
identify all object nodes that are used as parameters of the 
action, and we add data edges from those object nodes to 
the action node. Consider the instruction OOOf: invoke-virtual vO, 
Ljava/io/BufferedReader;.readLine:() Ljava/lang/String;, it has an input 
parameter, which is the register vO, using M current we know 
that when we create an action node from the instruction, vO is 
currenty holding a reference to a BufferedReader object node. 
We then add a data edge from the object node to the newly 
created action node. An instruction may return a value to a 
register. If it does not create a new object i.e. it returns a 
reference of an object has already been created before, then we 
identify the object node represent that object by M current and 
add a data edges from the action node to this node. Otherwise, 
we create a new object, add a data edge from the action node 
to the new object node and update mappings in M current . 

D. Sequence Extractor 

In this section, we describe how to extract API method 
sequences for one object or multiple objects from of a 
method using ARUS. For single object, sequence extractor 
scans through an ARUS to find object nodes associates with 
Android API objects. For each object node Oi we find all 
action nodes that have data edges connected with Oi and sort 
these action nodes by execution order. From the sorted action 
sequence, we only consider action nodes that represent API 
methods to get an API method sequence. For example, in 
Figure [8] for the first ARUS, there are three action nodes 
related to the object node BufferedReader and all of them 
are Android API methods, thus, the API method sequence 
associates the object node BufferedReader is (BufferedReader.init, 


BufferedReader.readLine, BufferedReader.close). 

To extract action sequences that involve multiple API 
objects, we first identity usage-dependent object sets. For 
each action node that represents API methods, we collect 
all API object nodes that have data-dependent with it, then 
we form the set of object types corresponds to those object 
nodes. In the first ARUS of Figure [8] there are two sets 
of usage-dependent objects: (FileReader, BufferedReader) has 
data-dependency with the action node BufferedReader.init, and 
(BuffedReader, String) has data-dependency with the action node 
BufferedReader.readLine.After collecting usage-dependent object 
sets, for each object set, we extract corresponding API method 
sequences using same technique for single object. For example, 
the API method sequence for the set (FileReader, BufferedReader) 
in the first ARUS is (FileReader.init, BufferedReader.init, Buffere¬ 
dReader.readLine, BufferedReader.close). 

There is one method sequence for an object (set of usage- 
dependent objects) in each ARUS of a method. Thus, after 
extracting method sequences in all ARUSs of the method, we 
remove duplicate sequences and only keep distinct sequences 
for each object (set of usage-dependent objects). 

E. HAPI Learner 

The HAPI learner uses a collection of API method se¬ 
quences of an object (a set of usage-dependent objects) to 
learn the API usages model. In this section, we describe the 
training algorithm to estimate parameters of HAPI model from 
training data. We also present a method to choosing the number 
of hidden states. 

1) Training Algorithm: The training algorithm aims to 
estimate HAPI’s parameters A = (A, B,n) given a collection 
of API method sequences. In general, Baum-Welch algorithm 
is often used to estimate parameters of Hidden Markov Model 
|7J. In this paper, we present a modified version of Baum- 
Welch algorithm for the problem of learning API usages. 
The input of the algorithm is a collection of API method 
sequences. We observed that there are many method sequences 
are duplicated in the colllection. Thus, to save space and speed 
up the training algorithm, we store training data as a map, 
where each method sequence is mapped to the number of 
time it occurs in the collection. Initially, the parameters of a 
HAPI are assigned with random values. The main idea of the 
algorithm is to iteratively estimates parameters to maximize the 
likelihood function (the probability of generating data given 
model). The iterative process terminates when the estimated 
values of parameters converge. 

Figure [9] shows the algorithm for training HAPI. its input 
includes training data S and the number of hidden states K. 
The parameters of the model are initialized randomly (line 1). 
Let us describe steps in the main loop of the algorithm: 

Step 1. Compute forward and backward probabilities. For 
each method sequence in training data we use dynamic pro¬ 
gramming to compute forward probabilities a l t (using Forward 
function) and backward probabilities /3,.t (using Backward func¬ 
tion) 171. a, t is defined as the probability of seeing partial 
method sequence yi, •••, Ut and being in state qi at time t 
given the model A: 

a itt = P(yi,y 2 ,...,y t ,it = q t \X) (1) 

/3i(t) is the probability of seeing the ending partial sequence 
yt- |-i, ...,2/r given state qi at time t and the model A: 

Pi,t = P(yt+i,-,yT\it = qt, A) 


( 2 ) 
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function TrainHAPI(TrainSet S, NHiddenStates K ) 
initialize A = ( A, B , it) with random values 
repeat 

foreach (Y n = (yi,y 2 , ■■■,y T ),c n ) e S 
a = Forward(A, Y n , T) 

/3 = Backward(A, Y n , l) 

for i e 1 ..K:// Compute state and state transition probabilities 

t 


7 ," (0 = „ 

*-‘j = 1 3,t p 3,t 

for j 6 G 1..T - 1: 
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for i e //update model parameters 
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e V: 
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until convergence 

return A 






function ForwardfHAPI A, APISequence Y, Position P) 

a - {a M } 1 < i < K,1 < t < P 
for i £ 1 ..K: cti.i = rribi{yi) 
for i 6 1..K, t e 2..P: 

<*i,t = bi(yt) Hf=i ai.t-ia.ij 

return a 

function Backward(HAPI A, APISequence Y, Position P) 

0 = {0i,t} l<i<K,P <t<T 
for i e 1..AT: 0 i<T = 1 
for i 6 1..K, t e T - 1..P: 

Pi.t — f2j=l Pj,t+l a ij bj (yt+l) 

return 8 


Fig. 9: Training Algorithm 



Fig. 10: HAPI for pair of objects FileReader and Buffered Reader 


In this method, we divide a training set in to two sets, one 
is used to train models and one is used as validation data 
to optimize the number of hidden states. For each K in a 
range, we train a HMM model with K as the number of 
hidden states. Then, for each model, we compute the likelihood 
of validation data which is the probability of generating the 
validation data given the model. We then choose K of the 
model that maximize the likelihood of the validation set. Figure 


10 shows graph representation the HAPI model for the pair 
of objects FileReader and BufferedReader as a result of HAPI 
learner component. To make to display, we round probabilities 
and ignore probabilities that is less than 0.01. 


Step 2. Compute state probabilities 7 and state transition 
probabilities £ using forward and backward probabilities. 7?(f) 
is probability of being at state q, at time t given method 
sequence Y n and the model A: 

7 i (t) = P(it=qt\Y n , A) = (3) 

The state transition probability (t) is the probability of being 
in state t ^ at time t and making transition to state q :j at time 
t + 1 given method sequence Y and the model A: 

Zij (f) = P(it = qi, it +1 = qj \ Y „, A) (4) 

Step 3. Reestimate model parameters. In this step, the model 
parameters are estimated as expected values of probabilities 
that we computed in the previous step. 7 "(f) is computed as 
the expected number of time state q, is at time 1. o^ +1 ^ is 
estimate as the expected number of transitions from state qi to 
state q t compared to the expected total number of transitions 
from state qj. bj S+1 \v m ) is the expected number of times the 
output method have been equal to v m while in state i over 
the expected total number of times in state qj. In the update 
equations, D is the total number of method sequences in the 
training set, and l yt — Vm is the indicator function. 

2) Choosing the number of hidden states: When training 
the HAPI model for an object (or a set of objects), we need 
to specify number of the hidden states as an input for the 
training algorithm, but we often do not know the how many 
states that the object has. Our idea in choosing the number 
of hidden states I\ is try to build models with different K 
and find a model that best describe new method sequences. 
This problem is equivalent to find K that maximizes the 
probability of generating new data, i.e. likelihood function. 


F. API Usage Recommendation 

In this section, we present an algorithm for recommending 
API method call using HAPI. The input of the algorithm is the 
HAPI model H of an object (set of objects) and the incomplete 
API method calls Y = (j/i,..., y.v) associated with the object 
(set of objects) in current editing code. The location T of Y 
is missing. The output of the suggestion algorithm is a ranked 
list of API methods that could be filled as the method call at 
position T. The idea of our algorithm is to place each API 
method as the method call of Y at location T and compute 
score of this assignment as the probability of generating the 
updated sequence (including the new API method) given the 
HAPI model. Then we add the API method with score to the 
ranked list of all candidates. 


1 function NextAPICallfHAPI A, APISequence Y, Location T) 

2 R = (D//a ranked list of candidates 

3 a = Forward(A, Y, T - 1) 

4 0 = Backward(A, Y.T + l) 

5 for v e V: 

6 for i e 1..K: 

7 au,T =» bi(v) J2f=i «i,T-iOy 

8 0i.T = EjLi 0j,T+iaijbj(v) 

9 score — 77, (Xi.T0i,T 

10 UpdateCandidateListfit, v, score) 

11 return it 

Fig. 11: Algorithm for suggesting the next API method call 

Figure 0 shows the algorithm. In the first part of our 
algorithm, we compute forward probabilities at position T — 1 
(using Forward function). We also compute backward probabili¬ 
ties at position T+ 1 (using Backward function). Then, we place 
each API method v as the method call of Y at position T and 
compute forward and backward probabilities at that position 
(line 6-8). The score of v is the probability of generating 
sequence (j/i, ...,Ut = v, ...,yjv) is computed by summing all 

























TABLE I: Data Collection 


Number of apps analyzed 22,330 

Number of classes analyzed 4,861,196 

Number of methods analyzed 9,117,943 

Number of bytecode instructions analyzed 258,726,894 

Space for storing apps’ bytecode (.dex files) 68 GB 


product of forward and backward probabilities: 

K K 

P{Y,yr = v|A) = ^P(V',y T = v,i T = <7;|A) = 

i =1 i =1 

The algorithm returns a ranked list of all the API method 
candidates with scores for suggestion. 

V. Empirical Evaluation 

In this section we conducted several experiments to eval¬ 
uate our approach. The first set of experiments aims to study 
the run-time performance of SALAD in extracting method 
sequences from bytecode and in training its API usage models. 
The second set aims to compare the effectiveness of those 
usage models to the baseline model ?r-gram in recommending 
API method calls. All experiments are executed on a computer 
running 64-bit Ubuntu 14.04 with Intel Core i5 CPU 3.2Ghz 
processor, 4GB RAM, and a 500 GB drive storage. 

A. Collecting Data 

Table [T] summarizes the data we collected for our experi¬ 
ments. In total, SALAD downloaded and analyzed 22,330 apps 
belonging to 26 categories from Google Play Store. For each 
category, we chose only top-ranked apps in the “Free” and 
“New Free” lists. This selection is based on the assumption 
that the top-ranked apps would have high quality code, and 
thus, would contain API usages of high interest for learning. 
We excluded apps in the “Game” category because they are 
often developed using game engines, thus, contain few actual 
API usages. 

Since Android mobile apps are distributed as .apk files, 
SALAD unpacked each downloaded apk file and kept only its 
.dex file, which contains all bytecode of the app. The total 
storage space for the .dex files of all downloaded apps is 
around 68 GB. SALAD parsed those dex files, which contain 
nearly 5 million classes. It analyzed each class and looked 
for methods implemented in that class to build ARUS models. 
Since an Android mobile app is self-contained, its .dex file 
contains bytecode of all external libraries it uses. That leads to 
the duplication of bytecode of shared libraries. Thus, SALAD 
maintains a dictionary of the methods it has already analyzed 
to help it to analyze each method only once. Because SALAD 
built ARUS for all possible execution paths, the number of 
ARUS could be exponential to the number of branching points 
in the code. Thus, to make the building process feasible, we 
excluded methods with more than 10 if instructions. We also 
excluded getter and setter methods, which are often short 
(less than 7 instructions) and do not contain API usages. The 
remaining consists more than 9 million methods which have 
in total nearly 260 million bytecode instructions. 

B. Extracting API Method Sequences 

SALAD built ARUS for each remaining method in the 
dataset and extracted API method sequences from those mod¬ 
els. It extracted sequences for both single object usages and 


TABLE II: Extracting API method sequences 



Single object 

Multiple object 

Number of distinct object types (sets) 
Total number of method sequences 
Average number of method sequences 
Average length of method sequences 

2,708 

17,973,832 

6,637 

3.8 

24,192 

20,611,024 

852 

8.0 

TABLE III: Training API usage models 


Single object 

Multiple object 

Number of trained models 

Average number of hidden states 

Total training time 

Total space to store trained models 

1,152 

8.1 

10 min 
7.2 MB 

7,228 

8.3 

70 min 

18.5 MB 


multiple object usages. Since we focus on learning Android 
API usages, only sequences involving classes and methods 
of Android application framework were extracted. Sequences 
with only one method call were disregarded. 

Table [D] summarizes this sequence extraction process. In 
total, SALAD has extracted nearly 18 millions API method 
sequences involving single object usages of more than 2,700 
different API object types. There are more than 24,000 dis¬ 
tinct usage-dependent object sets made from those types and 
SALAD has extracted over 20 millions method sequences for 
those sets (i.e. these sequences representing multiple object us¬ 
ages). The running time and space is acceptable: SALAD took 
9 hours to build ARUS and extract those method sequences 
(i.e. 1.5 seconds per app on average) and needed less than 300 
MB of working memory. 

C. Training API Usage Models 

Once all API method sequences are extracted and stored, 
SALAD trained HAPI models for each object type or usage- 
dependent object sets. For example, a HAPI is trained for the 
usages involving any MediaRecorder object and another HAPI 
is trained for the usages involving any two objects {FileReader, 
Buffered Reader}. We used only 80% of method sequences 
associated with each HAPI for training (of which 12.5% was 
randomly selected as the validation set). The 20% remaining 
was held out as testing data for the experiments with API 
recommendation tasks. To make the training process stable, 
we did not train HAPI models having insufficient method 
sequences (we used the cut-off threshold of 25 sequences). 
The training process is summarized in Table [HI] As seen, it is 
time- and space-efficient: More than 8,000 usage models were 
trained in about 80 minutes and stored using 26 MB. 

D. Sensitivity Analysis 

Table [III] suggests that HAPI models often have 8 internal 
states. To further study the effect of the number of hidden states 
K on the predictive power of HAPI models, we performed an 
extra experiment. In this experiment, we trained four HAPI 
models for both single object and multiple objects usages, 
varied K from 1 to 16 and measured the log-likelihood as 
their predictive power on validation data. 

Figure [12] illustrates the experiment results. As seen, the 
log-likelihood curves for four HAPI models have the same 
shape. When K is small (K = 1,2,3), the predictive power 
is low. This is reasonable because the models have too few 
hidden states to distinguish method sequences of different 
usage patterns. Then, when K increases, the predictive power 
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Fig. 12: Predictive power of HAPI models with different 
numbers of internal hidden states 

increases. However, after reaching a certain point, the predic¬ 
tive power starts to decrease because the hidden states might 
become redundant and overlap with each other, making the 
HAPI model overfit to the training data and under-performed 
on the validation data. 

The optimal number of hidden states of each model varies 
and is likely dependent on usage scenarios of the corre¬ 
sponding API objects. For example, as seen in Section II, an 
MediaRecorder has many different usage scenarios. However, 
the object pairs {FileReader, BufferedReader} are usually used 
for reading files with a dominant usage scenario illustrated in 
Figure [10] Thus, HAPI for MediaRecorder has more internal 
states than one for {FileReader, BufferedReader}. 

E. API Usage Recommendation 

In this experiment, we aim to measure the accuracy of 
HAPI models in recommending API usages. Following prior 
work, we design two different recommendation tasks. 

Task 1 - Predicting next call. An API method sequence is 
given and the model needs to recommend the most probable 
next call. This task has been used in 0.0 for the evaluation. 
Task 2 - Filling in a hole. The given API method sequence has 
a method call missing at a specified position, and the model 
needs to recommend the most probable call for that position. 
This task has been used in ED for evaluation. 

We chose n-gram model as the baseline for comparison 
due to several reasons. First, it is a statistical model, thus, 
is comparable to HAPI, which is also a statistical model. 
In addition, it is widely used in recent research on code 
completion 0, 0- Most importantly, n-gram model for 
API usages is studied by Raychev et al. 0 and reported as 
one of the state-of-the-art approaches. 

An n-gram model learns all possible conditional proba¬ 
bilities P(c|5'), where c is a method call and S' is a method 
sequence of n — 1 calls. This is the probability c occurs as 
the next method call of S. Using the chaining rule, we use an 
n-gram model to compute the generating probabilities of any 
given method sequence. In our experiment, we used a 3-gram 
model, i.e. the occurrence probability of a method call depends 
on its two previous calls. We trained a 3-gram model for each 



Fig. 13: Accuracy on Task 1 - Predicting next call 
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Fig. 14: Accuracy on Task 2 - Filling in a hole 


object type (or set) using the same training data as we trained 
HAPI models. Both models also use the same test data. 

1) Task 1 - Predicting next call: In this task, we evaluated 
all method calls in every sequence in the testing set. For 
a method call Ci at position i, we used its prefix of i — 1 
method calls as the input of a model and received its rec¬ 
ommendation of the top-fc most probable next method calls 
Rk = {?T, r2,..., rfc}. If a is in Rk, we consider it as an 
accurate recommendation (i.e. a hit). The top-/;- accuracy is 
the ratio of the total hits over the total number of evaluated 
method calls. 

Figure [13] shows the experiment results. As seen in the 
charts, the accuracy is consistent for both single and multiple 
object usages. More importantly, HAPI can recommend API 
usage for this task with a high level of accuracy. For example, 
for single object usage, it has a top-3 accuracy of 71.9% and 
a top-10 accuracy of around 90.5%. In addition, HAPI always 
outperforms n-gram model. For example, the top-3 and top-10 
accuracy results of n-gram model for single object usages are 
58.1% and 77.2%, respectively. The average improvement of 
HAPI over n-gram model across all settings is 12.6%. 

2) Task 2 - Filling in a hole: In this task, for each method 
sequence in a test set, we removed a method call at a random 
position (i.e. making a hole). Then, each available API method 
was substituted into that position and both HAPI and n- 
gram models were used to compute the probability of the 
entire sequence, respectively. Top-fc methods that made the 
k highest probabilities are selected as the recommendation. If 
the original method is one among them, it is counted as an 
accurate recommendation (i.e. a hit). The top-/;- accuracy is 
the ratio of the total hits over the total number of evaluated 
method sequences. 





























































Figure 14 shows the experiment results. As seen in the 
charts, the accuracy is consistent for both single and multiple 
object usages and is close to the accuracy for Task 1. The 
conclusions are also the same as for the experiment for Task 
1. That is, i) HAPI can recommend API usage for this task 
with a high level of accuracy and ii) HAPI always outperforms 
77 -gram model. For example, for single object usages, the top- 
3 and top-10 accuracy results of HAPI are 69.4% and 88.9%, 
while those of n-gram model are 54% and 74.1%, respectively. 
The average improvement of HAPI over n-gram model across 
all settings is 14.4%. 

The results suggests that for “Filling in the hole” task, 
HAPI model has higher improvement over n-gram model than 
for “Predicting next call” task. One explanation for this result is 
that in “Filling in the hole” task, HAPI has the whole sequence 
as the context for prediction, while in “Predicting next call” 
task, it has only the first part of the sequence. In both tasks, 
n-gram always uses only two prior calls. 


VI. Related Work 

There exist several works that proposed statistical models 
for learning API usages. Raychev et al. ED use n-gram and 
recurrent neural networks (RNN) to learn API usage patterns 
per object which are used to predict and suggest next API calls. 
Nguyen et al. ED presented GraLan, a graph-based statistical 
language model that learns common API usage (sub)graphs 
from a source code corpus and computes the probabilities of 
generating new usage graphs given the observed (sub)graphs. 

Statistical language models have been successfully used to 
capture patterns in source code. Hindle et al. ED showes that 
source code is repetitive and predictable like natural language 
and they adopted n-gram model on lexical tokens to suggest 
the next token. SLAMC ED represents code by semantic 
tokens, i.e. annotations of data types, method/field signatures, 
etc. rather than lexical tokens. SLAMC combines n-gram mod¬ 
eling of consecutive semantic tokens, topic modeling of the 
whole code corpus, and bi-gram of related API functions. Tu 
et al. 03 exploited the localness of source code and improve 
n-gram language model with caching for recently seen code 
tokens to improve next-token suggestion accuracy Allamanis 
and Sutton 03 trains n-gram language model a giga-token 
source code corpus. NATURALIZE 03 use n-gram language 
model to learns the style of a codebase and suggest natural 
identifier names and formatting conventions. Jacob et al. 03 
uses n-gram model to learn code templates. Hidden Markov 
Model has been used infer the next token from user-provided 
abbreviations 03 and detect coded information islands, such 
as source code, stack traces, and patches, from free tex |20|. 
Maddison et al. proposed tree-based generative models for 
source code. Hsiao et al. (22) learns n-gram language model 
on program dependence graph and uses the model for finding 
plagiarized code pairs. 

Pattern mining approaches represent usage patterns using 
various data structure such as sequences, sets, trees, and 
graphs. JADET |23j extracted a usage model in term of a finite 
state automaton (FSA). MAPO (24) mined frequent API call 
sequences and suggests associated code examples. Wang et al. 
(25) mines succinct and high-coverage API usage patterns from 
source code. Acharya et al. (26) proposed an approach to mine 
partial orders among APIs. Buse and Weimer (27) propose 
an automatic technique for mining synthesizing succinct and 


representative human-readable API examples. Other mining 
techniques includes mining associate mles (28) , item sets (29) , 
subgraphs |j3), (30), code idioms HD etc. 


One application of usage patterns mined from existing code 
is to support code completion. Grapacc (4| mines and stores 
API usage patterns as graphs and suggest these graphs in 
current editing code. Bruch et al. proposed three approaches 
for code completion. First, FreqCCS recommends the most 
frequently used method call. Second, ArCCS mines associate 
rules and suggest methods that often occur together. Finally, a 
best matching neighbors code completion technique that makes 
used fc-nearest-neighbor algorithm. SLANG ED uses n-gram 
to suggest the next API call based on a window of n — 1 
previous methods. Precise (32) mines existing code bases and 
builds a parameter usage database. Upon request, it queries 
the database and recommends API parameters. Graphite (33) 
allows library developers to introduce interactive and highly- 
specialized code generation interfaces that could interact with 
users and generates appropriate source code. 

Other approaches have been proposed to improve code 
completion tasks. Robbes et al. (34) improves code completion 
with program history. They measure the accuracy of replaying 
entire change history of programs with completion engine and 
gather information to improve the engine. Hou and Pletcher 
(35) found that ranking method calls by frequency of past use 
is effective and propose new strategies for organizing APIs in 
the code completion pop up. Hill and Rideout |36| matches 
the fragment under editing with small similar-structure code 
segments that frequently exist in large software projects. 
McMillan et al. (37 [ and Subramanian et al. (38) use API 
documentation to suggest source code examples to developers. 
Holmes and Murphy (39) describe an approach for locating 
relevant code examples based on heuristically matching with 
the structure of the code under editing. 


VII. Conclusion 

We propose a statistical approach to learn API usages 
from bytecode of Android mobile apps. We propose a graph- 
based representation of usage scenarios to extract API method 
sequences from bytecode and use those sequences to learn API 
usage models based on Hidden Markov Models. Our empirical 
evaluation on a large dataset indicates that our approach can 
learn useful API usage models which can provide higher levels 
of accuracy in API recommendation than the n-gram model. 
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